The current reinforcement learning algorithm uses forward-generated trajectories to train the agent. The forward-generated trajectories give the agent little guidance, so the agent can explore as much as possible. While the appreciation of reinforcement learning comes from enough exploration, this gives the trade-off of losing sample efficiency. The sampling efficiency is an important factor that decides the performance of the algorithm. Past tasks use reward shaping techniques and changing the structure of the network to increase sample efficiency, however these methods require many steps to implement. In this work, we propose novel reverse curriculum reinforcement learning. Reverse curriculum learning starts training the agent using the backward trajectory of the episode rather than the original forward trajectory. This gives the agent a strong reward signal, so the agent can learn in a more sample-efficient manner. Moreover, our method only requires a minor change in algorithm, which is reversing the order of trajectory before training the agent. Therefore, it can be simply applied to any state-of-art algorithms.

在这项工作中，我们研究了解决强化学习问题的基于政策的方法，其中采用了非政策性采样和线性函数近似进行政策评估，以及包括自然政策梯度（NPG）在内的各种政策更新规则，用于政策更新。为了在致命三合会的存在下解决政策评估子问题，我们提出了一个通用算法的多步型TD学习框架，具有广义的重要性抽样比率，其中包括两个特定的算法：$ \ lambda $ Q Q $ Q Q $ - 跟踪和双面$ Q $ - 跟踪。通用算法是单个时间尺度，具有可证明的有限样本保证，并克服了非政策学习中的高方差问题。至于策略更新，我们仅使用Bellman操作员的收缩属性和单调性属性提供通用分析，以在各种策略更新规则下建立几何融合。重要的是，通过将NPG视为实施政策迭代的近似方法，我们在不引入正则化的情况下建立了NPG的几何融合，并且不使用现有文献中的镜像下降类型的分析类型。将策略更新的几何融合与策略评估的有限样本分析相结合，我们首次建立了整​​体$ \ Mathcal {o}（\ Epsilon^{ - 2}）$样本复杂性以找到最佳策略（最多达到函数近似误差）使用基于策略的方法和线性函数近似下的基于策略的方法。

由于众所周知，强化学习算法是数据密集型的，因此从环境中进行采样观测的任务通常在多个代理之间分配。但是，将这些观察结果从代理转移到中心位置可能会非常昂贵，并且还可以损害每个代理人本地行为政策的隐私。在本文中，我们考虑了一个联合加强学习框架，其中多个代理商协作学习了一个全球模型，而无需共享他们的个人数据和政策。每个代理都维护模型的本地副本，并使用本地采样数据对其进行更新。尽管具有n个代理可以启用n次数据的采样，但尚不清楚它是否导致比例收敛的加速。我们提出了联合版本的On-Policy TD，Off-Policy TD和Q学习，并分析其收敛性。对于所有这些算法，据我们所知，我们是第一个考虑马尔可夫噪声和多个局部更新的人，并证明相对于代理的数量是线性收敛的速度。为了获得这些结果，我们表明联邦TD和Q学习是与马尔可夫噪声联合随机近似的一般框架的特殊情况，并且我们利用该框架提供了适用于所有算法的统一收敛分析。

随机近似（SA）和随机梯度下降（SGD）算法是现代机器学习算法的工作马。由于快速收敛行为，它们在实践中优选它们的持续步骤变体。然而，恒定的步骤随机迭代算法不与最佳解决方案渐近地收敛，而是具有静止分布，这通常不能被分析表征。在这项工作中，我们研究了适当缩放的静止分布的渐近行为，在恒定步骤零的限制中。具体而言，我们考虑以下三种设置：（1）SGD算法，具有平滑且强的凸面物镜，（2）涉及Hurwitz矩阵的线性SA算法，和（3）涉及收缩算子的非线性SA算法。当迭代以$ 1 / \ sqrt {\ alpha} $缩放时，其中$ \ alpha $是常量的步骤，我们表明限制缩放静止分布是整体方程的解决方案。在该等式上的唯一性假设（可以在某些设置中除去），我们进一步表征了作为高斯分布的限制分布，其协方差矩阵是合适的Lyapunov方程的独特解决方案。对于超出这些情况的SA算法，我们的数值实验表明，与中央极限定理类型结果不同：（1）缩放因子不需要为$ 1 / \ sqrt {\ alpha} $，并且（2）限制分布不需要高斯。基于数值研究，我们提出了一种确定右缩放因子的公式，并与近似随机微分方程的欧拉 - 玛赖山离散化方案进行富有洞察力的连接。

The decarbonization of buildings presents new challenges for the reliability of the electrical grid as a result of the intermittency of renewable energy sources and increase in grid load brought about by end-use electrification. To restore reliability, grid-interactive efficient buildings can provide flexibility services to the grid through demand response. Residential demand response programs are hindered by the need for manual intervention by customers. To maximize the energy flexibility potential of residential buildings, an advanced control architecture is needed. Reinforcement learning is well-suited for the control of flexible resources as it is able to adapt to unique building characteristics compared to expert systems. Yet, factors hindering the adoption of RL in real-world applications include its large data requirements for training, control security and generalizability. Here we address these challenges by proposing the MERLIN framework and using a digital twin of a real-world 17-building grid-interactive residential community in CityLearn. We show that 1) independent RL-controllers for batteries improve building and district level KPIs compared to a reference RBC by tailoring their policies to individual buildings, 2) despite unique occupant behaviours, transferring the RL policy of any one of the buildings to other buildings provides comparable performance while reducing the cost of training, 3) training RL-controllers on limited temporal data that does not capture full seasonality in occupant behaviour has little effect on performance. Although, the zero-net-energy (ZNE) condition of the buildings could be maintained or worsened as a result of controlled batteries, KPIs that are typically improved by ZNE condition (electricity price and carbon emissions) are further improved when the batteries are managed by an advanced controller.

The emergence of large pretrained models has enabled language models to achieve superior performance in common NLP tasks, including language modeling and question answering, compared to previous static word representation methods. Augmenting these models with a retriever to retrieve the related text and documents as supporting information has shown promise in effectively solving NLP problems in a more interpretable way given that the additional knowledge is injected explicitly rather than being captured in the models' parameters. In spite of the recent progress, our analysis on retriever-augmented language models shows that this class of language models still lack reasoning over the retrieved documents. In this paper, we study the strengths and weaknesses of different retriever-augmented language models such as REALM, kNN-LM, FiD, ATLAS, and Flan-T5 in reasoning over the selected documents in different tasks. In particular, we analyze the reasoning failures of each of these models and study how the models' failures in reasoning are rooted in the retriever module as well as the language model.

Quantum state tomography aims to estimate the state of a quantum mechanical system which is described by a trace one, Hermitian positive semidefinite complex matrix, given a set of measurements of the state. Existing works focus on estimating the density matrix that represents the state, using a compressive sensing approach, with only fewer measurements than that required for a tomographically complete set, with the assumption that the true state has a low rank. One very popular method to estimate the state is the use of the Singular Value Thresholding (SVT) algorithm. In this work, we present a machine learning approach to estimate the quantum state of n-qubit systems by unrolling the iterations of SVT which we call Learned Quantum State Tomography (LQST). As merely unrolling SVT may not ensure that the output of the network meets the constraints required for a quantum state, we design and train a custom neural network whose architecture is inspired from the iterations of SVT with additional layers to meet the required constraints. We show that our proposed LQST with very few layers reconstructs the density matrix with much better fidelity than the SVT algorithm which takes many hundreds of iterations to converge. We also demonstrate the reconstruction of the quantum Bell state from an informationally incomplete set of noisy measurements.

Automation in farming processes is a growing field of research in both academia and industries. A considerable amount of work has been put into this field to develop systems robust enough for farming. Terrace farming, in particular, provides a varying set of challenges, including robust stair climbing methods and stable navigation in unstructured terrains. We propose the design of a novel autonomous terrace farming robot, Aarohi, that can effectively climb steep terraces of considerable heights and execute several farming operations. The design optimisation strategy for the overall mechanical structure is elucidated. Further, the embedded and software architecture along with fail-safe strategies are presented for a working prototype. Algorithms for autonomous traversal over the terrace steps using the scissor lift mechanism and performing various farming operations have also been discussed. The adaptability of the design to specific operational requirements and modular farm tools allow Aarohi to be customised for a wide variety of use cases.

Communication enables agents to cooperate to achieve their goals. Learning when to communicate, i.e., sparse (in time) communication, and whom to message is particularly important when bandwidth is limited. Recent work in learning sparse individualized communication, however, suffers from high variance during training, where decreasing communication comes at the cost of decreased reward, particularly in cooperative tasks. We use the information bottleneck to reframe sparsity as a representation learning problem, which we show naturally enables lossless sparse communication at lower budgets than prior art. In this paper, we propose a method for true lossless sparsity in communication via Information Maximizing Gated Sparse Multi-Agent Communication (IMGS-MAC). Our model uses two individualized regularization objectives, an information maximization autoencoder and sparse communication loss, to create informative and sparse communication. We evaluate the learned communication `language' through direct causal analysis of messages in non-sparse runs to determine the range of lossless sparse budgets, which allow zero-shot sparsity, and the range of sparse budgets that will inquire a reward loss, which is minimized by our learned gating function with few-shot sparsity. To demonstrate the efficacy of our results, we experiment in cooperative multi-agent tasks where communication is essential for success. We evaluate our model with both continuous and discrete messages. We focus our analysis on a variety of ablations to show the effect of message representations, including their properties, and lossless performance of our model.

Syntax is a latent hierarchical structure which underpins the robust and compositional nature of human language. An active line of inquiry is whether large pretrained language models (LLMs) are able to acquire syntax by training on text alone; understanding a model's syntactic capabilities is essential to understanding how it processes and makes use of language. In this paper, we propose a new method, SSUD, which allows for the induction of syntactic structures without supervision from gold-standard parses. Instead, we seek to define formalism-agnostic, model-intrinsic syntactic parses by using a property of syntactic relations: syntactic substitutability. We demonstrate both quantitative and qualitative gains on dependency parsing tasks using SSUD, and induce syntactic structures which we hope provide clarity into LLMs and linguistic representations, alike.